This disclosure relates to the capture of color motion imagery, and specifically to the capture of color motion imagery by sequential frame samples of different color filtration, combined with a sensor having a color filter array overlaid on its imaging pixels.
There are three basic approaches to capturing color motion imagery. The first uses three sensors, typically using red, green and blue filters, which simultaneously capture a scene""s dynamic content. This technique is used with both tube pickup devices and with two dimensional sensor arrays, such as charge coupled devices (CCD) or composite metal-on-silicon (CMOS) devices, which are referred to as 3-CCD cameras.
The second approach uses a single two dimensional (2D) sensor having the color filtration applied separately to each pixel. Colors are arranged in spatially varying patterns that are designed to provide a high spatial bandwidth for luminance or green, and to minimize color aliasing artifacts. The result is that each color layer has incomplete samples per frame but special interpolation algorithms are used to reconstruct full dimensioned frames for each color layer. This approach is known as color filter array (CFA) camera capture.
The third approach is to use a single sensor with either no filtration, or uniform filtration across picture elements, and to combine this with a method to change the filtration over the whole sensor as a function of time. The idea is to temporally change the sensor filtration at rates faster than the temporal response of the eye, so that the sequential color lag, aka color breakup, is below visual threshold. It is most commonly referred to as field sequential color capture (FSC).
The primary disadvantage of the 3-CCD, or 3-tube, approach is the cost of three sensors. A second disadvantage is the problem of color mis-registration between the three sensors because of alignment in the optical path relative to the scene, which may impose tight manufacturing tolerances that increase manufacturing costs. Color mis-registration may cause luminance blur in textures with very small amounts of mis-registration, and may cause color bleeding, aka color fringing, at both achromatic and chromatic edges. If the registration is well aligned, this approach achieves the resolution of the sensors for all three color layers of a frame. Because of cost, however, this approach is only used for high-end studio video cameras, and digital still cameras designed for the professional and advanced hobbyist.
While the CFA approach is less expensive because of the use of a single sensor, it has many disadvantages. These include reduced spatial resolution, the necessity for an interpolation algorithm to reconstruct the three color frames for display, and the necessity for an anti-aliasing filter to prevent diagonal luminance high spatial frequencies from aliasing into lower frequency color patterns. Additionally, the color high spatial frequencies may alias into luminance or color patterns. Consequently, there is a trade-off between sharpness and color artifacts, which show up strongly in such common image content as highlight reflections in eyes, as well as the expected luminance high spatial frequencies such as in texture, e.g., hair, or geometric patterns. In current implementations, fairly complex interpolation algorithms that include pattern recognition are used in an attempt to maximize sharpness and minimize color spatial artifacts. Most cameras opt to avoid any chromatic aliasing because it is a new categorical distortion and favor the sharpness reduction, which is already present to some extent. In summary, CFA systems do not achieve the resolution of their sensor dimensions, either in luminance or in color.
The field sequential color technology is not presently a very active area, and much of the key work was done prior to the NTSC color standard, when field sequential color was a viable competitor for color television. Analog field sequential color video was difficult to accomplish at high frame rates. Its primary application was in telecine and other specialized applications. Recently, FSC activity increased because of full digitization of video systems. Digital video facilitates field sequential color capture to simultaneous color displays.
U.S. Pat. No. 2,738,377, to Weighton, granted Mar. 13, 1956 for Color Television, describes a color television system which uses a rotating color filter wheel, with equally spaced wedges in the order GRGBGRGB, with a single pickup tube, and a CRT to constitute a full color television system. The reference describes an eight-fold interlace, with the color wheel spinning fast enough to cause a different color filter for each interlace line. A sufficient number of lines are used to result in a captured image with 400 G lines and 200 R and B lines. The main difficulty with this approach is the extreme demand imposed on the system bandwidth due to the eightfold interlace. Another is that color field sequential display is required in this system, and because of eye movements, such displays are more susceptible to color breakup than color field sequential capture.
U.S. Pat. No. 3,604,839 to Kitsopoulos, granted Sep. 14, 1971 for Field-sequential color television apparatus employing color filter wheel and two camera tubes, describes a television system having a color filter wheel and two camera tubes, which is primarily aimed at telecine. The purpose of the two tubes is to allow for simultaneous capture of two different colors, thus allowing the exposure to lengthen, given the field rate constraints, and increase the signal to noise ratio.
U.S. Pat. No. 3,969,763 to Tan, granted Jul. 13, 1976, for Color television camera provided with a pickup device and color filter placed in front thereof, describes a color field sequential camera that uses a single pickup tube and a color filter wheel with many fine color strips. The purpose is to capture the color filtered sections of an image more spatially and temporally coincident. It approaches a color sequential interlace, and is primarily an analog hardware system addressing various delay and converter steps. The reference is notable because it discloses a system which uses liquid filters that are electronically controllable, rather than using a mechanical wheel. The system also captures in a YRB space, rather than the more common RGB.
U.S. Pat. No. 3,971,065 to Bayer, granted Jul. 20, 1976, for Color Imaging Array, describes the Bayer pattern color filter array.
U.S. Pat. No. 4,067,043, to Perry, granted Jan. 3, 1978 for Optical conversion method, describes the use of electro-optically controlling color filtration, in a field sequential mode, via the use of crossed polarizers.
U.S. Pat. No. 4,605,956 to Cok, granted Aug. 12, 1986, for Single-Chip Electronic Color Camera with Color-Dependent Birefringent Optical Spatial Filter and Red/Blue Signal Interpolating Circuit, describes an RGB CFA system, including birefringent optical prefilter and the interpolation algorithms for the subsampled R and B image layers.
U.S. Pat. No. 4,670,777 to Ishikawa et al., granted Jun. 2, 1987, for Colorfilter array having cyan, yellow, green, and magenta filter elements providing increased dynamic range for use with field integrating mode solid state imaging device, describes the use of subtractive color filters as well as an additive. The field integration refers to interlace fields, not color shutter fields.
U.S. Pat. No. 4,851,899 to Yoshida et al., granted Jul. 25, 1989, for Field-sequential color television camera, describes an RGB field sequential camera using a color wheel that causes the field order to be R1R2G1G2B1B2, etc. Because the first field of each color pair has a residual charge from the previous, different color, the system discards the first captured field in order to prevent color desaturation.
U.S. Pat. No. 4,967,264 to Parulski et al., granted Oct. 30, 1990, for Color sequential optical offset image sampling system, combines color field sequential concepts with sensor dithering to increase resolution. A color filter wheel is used containing, in order, 2 green, one red, and one blue element. The two green elements are placed at various tilts to create spatial offsets on the sensor, so the two green fields may be combined to create an image with higher green resolution than would be available solely from the sensor.
U.S. Pat. No. 5,084,761 to Nitta, granted Jan. 28, 1992, for Video camera with rotatable colorfilter and registration correction, addresses the color mis-registration caused by the variations in tilt and thickness of the color filter wheel in color field sequential cameras. The horizontal and vertical deflection signals are adjusted. Even though the color field sequential camera is, by design, not supposed to have a color registration problem, it does in fact have this problem because of the mechanics of the color wheel. The reference seeks to avoid this problem with electronically controlled color filters, such as a LCD.
U.S. Pat. No. 5,548,333 to Shibazaki et al., granted Aug. 20, 1996, for Color mixing prevention and color balance setting device and method for afield-sequential color television camera, addresses color mixing and color balance processes in a RGB filter wheel field sequential color camera. It allows a video field with a mixed color input, that is a transition from one color sector to another of the color wheel, to be used, but discards the signal during the mixed color time interval.
U.S. Pat. No. 5,631,703 to Hamilton et al., granted May 20, 1997, for Particular pattern of pixels for a color filter array which is used to derive luminance and chrominance values, describes a particular array using YMC and G.
U.S. Pat. No. 5,748,236, to Shibazaki, granted May 5, 1998, for Color mixing prevention and color balance setting device and method for afield-sequential color television camera, is based on a continuation-in-part application from the ""333 patent, supra, however, this reference concentrates on the hardware, and specifically, the concept of discarding the accumulated charge during the mixed interval.
U.S. Pat. No. 5,751,384 to Sharp, granted May 12, 1998, for Colorpolarizers for polarizing an additive color spectrum along a first axis and its compliment along a second axis, describes switchable color filter via polarization and LCD as would be used in digital still and video cameras.
U.S. Pat. No. 5,767,899 to Hieda et al., granted Jun. 16, 1998, for Image pickup device, addresses the problem of constant luminance, where the luminance signal is formed prior to any gamma correction. This is opposed to conventional video processing, where the gamma correction nonlinearity is applied to the RGB values prior to the formation of the luminance signal. In this device the captured Y, Cr, and Cb outputs from a CFA interpolation process are converted to RGB, gamma corrected, then converted to conventional YR-YB-Y. Various base-clipping, or coring, limiting, and color suppression based on the luminance signal are then incorporated.
A number of literature references in the field of human visual system modelling are relevant to invention. There are three basic visual properties relating the luminance channel, as represented by video Y, and the opponent color channels, as approximated by the color difference signals U and V. These are:
1. The maximum temporal frequency response of the opponent color system is less than xc2xd that of the luminance.
2. The maximum spatial frequency response of the opponent color system is near xc2xd that of the luminance.
3. The maximum sensitivity of opponent color system is slightly greater than xc2xd that of the luminance.
Properties #1 and #2 are best summarized in A. B. Watson Perceptual components architecture, JOSA A V. 7#10, 1990, pp1943-1954; while property #3 is described in K. T. Mullen The contrast sensitivity of human colour vision to red-green and blue-yellow chromatic gratings J. Physiol. V. 359, 1985, pp. 381-400. These three properties may be used in the design of the relative durations of exposures in order to prevent color lag and in the spatio-temporal integration following image capture.
There have also been recent vision science studies, namely L. Arend et al., Color breakup in sequentially scanned LCDs. SID 94 Digest, 1994, pp 201-204, and D. Post et al., Predicting color breakup onfield-sequential displays, SPIE Proc. V. 3058, 1997, pp 57-65, that specifically investigated color break-up, or color lag. Although these studies were performed for the application of field sequential color displays, some of their findings are relevant to color field sequential capture. One of these findings is that color lag detection has a strong luminance component. In fact, as the color offset between R, G and B layers is increased from zero to detectability, the first signal to exceed threshold is the luminance. This manifests itself as blurring in texture before any high contrast edge blur occurs, because of the masking effects caused by high contrast edges. Eventually, as the offset is increased, color artifacts become visible at high contrast edges.
Based on these findings, isolating the offsets of the luminance from the chrominance layers of an image is beneficial, as is maximizing the spatial capture of G, or luminance, Y, over that of R and B, or R-Y and B-Y. The work done with YRB camera systems, and documented in Hunt, The reproduction of colour in photography, printing, and television, 4th edition, 1987, pp 409-410, Fountain Press, England, is instructive. This work was investigated for three-sensor systems in the days when the sensors where tubes. The RYB system was developed in an attempt to reduce the visibility of the color offset problem due to manufacturing tolerances of optically aligning the tubes. Due to complications with gamma correction, it was abandoned in favor of RGBY systems, using four tubes.
Other works by H. Nabeyama et al, All-solid-State Color Camera with Single Chip MOS Imager IEEE Trans. On Consumer Electronics. V. CE-27, 1981, pp. 40-45; and T. Tanaka et al., IEEE Trans. On Consumer Electronics. V. CE-36, 1990, pp. 479-485, which describes the famous Y-C-G-M CFA, are relevant to the invention described herein.
A method of field sequential color image capture includes: optically capturing a scene frame-by-frame; filtering the scene through an active color shutter to produce a first color component set having plural first color components therein, thereby modifying the spectral transmission of the scene as a function of time; detecting the scene with a color filter array area sensor, wherein each first color component of the first color component set of the scene is detected at different points in time within a frame; dividing the detected into a second color component set having plural second color components therein; aligning the second color component set in time for each frame interval; storing each second color component in a memory unit; combining the stored second color components into a frame image; and processing the frame image for color reproduction and format.
A system for capturing a field sequential color image, includes an optical capture mechanism for capturing a scene frame-by-frame in frame intervals which operates at a predetermined shutter speed; a first color shutter for producing a first color component set having plural first color components therein, thereby modifying the spectral transmission of the scene as a function of time; a color filter array area sensor for sensing and modifying the first color components, wherein each first color component of the scene is detected at a different point in time within a frame; an array field selector for dividing the scene into a second color component set having plural second color components therein; a frame clock operating at a first predetermined rate, a color field control clock, operating at a second predetermined rate under control of the frame clock, for controlling the active color shutter and the area sensor at a second predetermined rate, multiple memory locations for storing each color component; and a field-to-frame combiner for combining the stored second color components into a frame image.
An object of the invention is to provide a system and method of color image capture using a CFA-FSC hybrid.
An object of the invention is to provide a system and method of increasing the temporal bandwidth of R and B color components by sacrificing some of the spatial bandwidth of the R and B layers, and to accomplish same without reducing the spatial or temporal bandwidth of the G, or luminance layer.